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Abstract 


Assessment  of  complex  cognitive  tasks  requires  an 
tmderstanding  of  the  characteristics  of  expertise  in  the  specific  domain. 
The  research  reported  in  this  paper  was  part  of  a  larger  project  whose 
goal  was  to  distinguish  among  levels  of  expertise  in  digital  circuit 
design.  This  study  examined  the  problem  solving  characteristics  of 
seven  electrical  engineering  students,  five  imdergraduates  and  two 
advanced  graduate  students.  They  were  asked  to  design  a  simple 
combinational  logic  digital  circuit.  This  type  of  circuit  can  be  designed 
using  a  standard  procedure  and  set  of  components  but  there  are  issues 
of  optimization  that  enhance  a  standard  design.  Subjects  provided 
think-aloud  protocols  during  the  solution  process.  Videotapes  and  the 
handwritten  artifacts  created  during  the  design  process  were  coded  for 
standard  components,  optimization,  and  discrete  behavioral  episodes,  a 
subset  of  which  defined  Planning-Related  Activity.  Contrary  to 
previous  contrastive  analyses  of  expertise,  all  subjects  showed  evidence 
of  planning.  However,  the  function  and  location  of  the  plaiming 
differentiated  among  subjects.  Global  planning,  planning  associated 
with  moving  from  one  component  to  the  next,  and  selecting  among 
alternatives  were  associated  with  better  circuit  designs.  Local  planning 
dominated  the  problem  solving  of  the  less  expert  designers. 
Implications  for  the  assessment  of  expertise  are  discussed. 
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Introduction 

The  knowledge  and  problem-solving  processes  that  support  the 
execution  of  complex  cognitive  skills  and  performances  have  been  the 
subject  of  cognitive  research  for  several  decades.  A  focus  of  this 
research  has  been  on  characterizing  differences  between  individuals 
who  are  experts  and  those  who  are  novices.  Such  contrastive  analyses 
have  been  undertaken,  in  part,  to  elucidate  the  "end  state" 
performance  goals  of  acquiring  cognitive  skills  and,  in  part,  in  hopes  of 
tmderstanding  how  to  assist  individuals  in  attaining  these  end  states. 
Researchers  have  taken  this  approach  in  a  number  of  domains 
including  physics,  chess,  architecture,  mechanical  engineering  and 
instructional  design  (e.g..  Akin,  1988;  Chase  &  Simon,  1973;  Chi, 
Feltovich,  &  Glaser,  1981;  Foz,  1973;  Goel  &  Pirolli,  1989). 

The  present  research  applies  the  expert-novice  contrastive 
method  to  the  processes  involved  in  the  design  of  combinational  logic 
digital  circuits.  More  specifically,  this  paper  reports  on  the  problem 
solving  that  occurred  during  the  solution  of  a  circuit  design  problem 
involving  combinational  logic.  The  rationale  for  this  focus  is  based  on 
(a)  previous  researchers'  reports  of  the  differences  between  experts  and 
novices  in  other  cognitive  domains,  and  (b)  a  more  broadly  based 
analysis  of  experts'  and  students'  problem-solving  activity  in  designing 
combinational  logic  circuits.  These  areas  are  reviewed  in  the  next  two 
sections  of  this  paper. 

Characteristics  of  Expertise 

Previous  research  has  resulted  in  some  general  characterizations 
of  the  attributes  of  experts  and  novices  in  terms  of  both  overall 
approach  and  specific  strategies  (e.g.,  Chi,  Glaser,  &  Farr,  1988;  Ericsson 
&  Smith,  1991;  Larkin,  1980).  Table  1  summarizes  the  major  findings. 
Several  differences  between  expert  and  novices  are  noteworthy.  For 
example,  Chi,  Feltovich  and  Glaser  (1981)  found  that  experts  represent 
problems  in  their  domain  at  a  deeper  level  than  novices.  Paige  and 
Simon  (1966)  showed  that  experts  spend  more  time  analyzing  a 
problem  qualitatively.  Experts  perceive  large  meaningful  patterns  in 
their  domains  of  expertise  (Akin,  1980;  Chase  &  Simon,  1973).  In 
addition,  several  researchers  fovmd  that  experts  have  strong  self 
monitoring  skills;  and  in  some  domains,  experts  solved  problems 
faster  than  novices  and  with  little  error  (Chase,  1983;  Gentner,  1988). 
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Insert  Table  1  about  here 


The  Design  Process 

There  have  been  a  number  of  studies  of  the  design  process  in 
architecture  and  in  various  areas  of  engineering  (e.g..  Akin,  1988;  Foz, 
1973;  Goel  &  Pirolli,  1989;  Vandermolen,  James,  Goldman,  Biswas,  & 
Bhuva,  1992).  Although  design  appears  to  be  a  highly  constructive  and 
possibly  idiosyncratic  process,  there  are  a  number  of  general  principles 
and  heuristics  that  students  of  design  are  taught  (e.g.,  Garrod  &  Boms, 
1991).  As  in  other  areas  of  complex  cognitive  activity,  researchers  have 
attempted  to  characterize  the  differences  between  individuals  who 
have  acquired  different  levels  of  design  skill. 

A  study  by  Akin  (1988)  studied  the  design  process  in  the 
architectural  domain.  Akin  compared  the  design  behaviors  of  6 
professionals  who  were  non-architects^,  6  students  of  architecture,  and 
6  expert  architectural  designers.  Their  task  was  to  design  four 
functional  areas  (a  conference  room,  a  chief  engineer's  room,  a  room 
for  two  staff  engineers,  and  a  secretarial  area)  in  three  differently 
shaped  sites:  square,  rectangle  and  L-shaped.  Each  subject  solved  the 
layout  problem  for  the  three  sites.  The  design  behaviors  were  studied 
using  a  think-aloud  protocol  method.  In  addition  to  the  verbalizations, 
texts  and  diagrams  generated  by  the  subjects  were  used  in  the  analysis. 

Akin  (1988)  concluded  that  experienced  designers  did  more 
restructuring  of  problems  than  novices  (professionals  who  were  non¬ 
architects  and  students  of  architecture),  that  fewer  conflicts  were 
present  in  the  tentative  solutions  of  experts,  and  that  modifications 
(backtracking)  by  experts  were  consistent  with  the  type  of  conflict,  e.g., 
local  alternatives  for  local  conflicts  and  global  alternatives  for  global 
conflicts.  Experts  were  also  found  to  use  scenario-like  constmcts  to 
represent  knowledge  about  a  given  functional  type.  Scenarios  are 
representations  of  "malleable  geometric  relationships"  between  the 
functional  imits;  the  geometric  parameters  of  the  scenarios  can  be 
modified  to  generate  new  alternatives. 

Non-architects  seemed  to  rely  on  prototypical  situations  familiar 
to  them,  in  contrast  to  the  more  irmovative  approaches  taken  by  the 
experts.  Non-architects  tended  to  use  actual  physical  templates.  The 


^Akin  does  not  clearly  identify  these  subjects'  domains  of  expertise. 
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students  relied  on  performance  evaluation,  i.e.,  they  assembled  their 
solutions  from  individual  analytical  observations  about  the  way  each 
partial  solution  performed  in  terms  of  each  problem  constraint.  Thus, 
the  approach  taken  by  the  students  failed  to  take  advantage  of  known 
solution  patterns  and  did  not  resolve  as  many  constraints  as  possible. 

In  another  domain,  machining,  Saiz  and  Breuleux  (1992) 
examined  planning,  writing,  and  evaluation  processes  among  12 
experts  and  12  novices  on  two  design  tasks  of  increasing  complexity. 
They  found  significant  differences  in  the  amount  of  evaluation 
performed  by  subjects,  with  novices  making  more  evaluation 
statements  than  experts;  they  also  found  that  the  amoimt  of  writing 
foimd  in  subjects'  protocols  decreased  on  the  more  complex  task.  With 
respect  to  amoimt  of  planning,  they  found  no  significant  effect  of  task 
complexity  and  no  significant  differences  between  experts  and  novices. 
However,  Saiz  and  Breuleux's  analysis  of  planning  may  not  have  been 
sensitive  enough  to  characterize  differences  in  types  of  planning.  An 
analysis  of  the  type  of  planning  rather  than  amount  might  have  been 
more  interesting. 

For  example,  Goel  and  Pirolli  (1989)  studied  three  different 
design  contexts:  architecture,  mechanical  engineering,  and 
instructional  design.  They  collected  a  total  of  12  think-aloud  protocols. 
From  this  database,  the  protocols  of  three  subjects,  one  from  each 
domain,  were  discussed.  The  three  subjects  varied  in  experience:  ten 
years  design  experience  in  designing  industrial  training  material,  six 
years  professional  experience  as  an  architect,  and  limited  design 
experience  in  mechanical  engineering. 

From  the  protocol  analysis,  Goel  and  Pirolli  identified  eight 
significant  invariants  in  the  problem  spaces  of  the  subjects  in  the  three 
different  design  disciplines: 

-  extensive  problem  structuring, 

-  extensive  performance  modeling, 

-  personalized  or  institutionalized  evaluation  functions  and 
stopping  rules, 

-  a  limited  commitment  mode  control  strategy, 

-  constraint  propagation, 

-  the  role  of  abstractions  in  the  transformation  of  goals, 

-  the  use  of  artificial  symbol  systems, 

-  solution  decomposition  into  leaky  modules^. 


^Leaky  modules  are  not  independent;  a  decision  made  in  one  module  could  have 
coruequences  in  several  others. 
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These  invariants  are  generally  consistent  with  the  results  of  other 
researchers  and  describe  design  behaviors  that  are  consistent  across 
different  domains. 

The  purpose  of  the  research  reported  in  this  paper  was  to  extend 
the  characterization  of  expertise  to  the  area  of  simple  circuit  design. 

This  is  an  interesting  domain  because  simple  circuit  design  tasks  can  be 
executed  either  as  a  set  of  standard  procedures  or  as  an  open-ended 
problem  that  can  be  approached  from  many  different  angles.  In  order 
to  find  an  optimal  solution,  the  designer  has  to  take  into  account  trade¬ 
offs  between  several  parameters.  Examples  of  such  parameters  in 
digital  circuit  design  are:  the  chip  area  that  the  circuit  will  occupy, 
delay  between  input  and  output  signals,  and  power  consumption  of  the 
circuit.  The  standard  approach  will  yield  a  solution  but  may  ignore 
interactions,  and  does  not  consider  these  trade-offs.  For  such  problems 
a  number  of  alternative  solutions  may  be  possible,  each  of  which 
satisfies  the  functional  requirements,  but  only  one  of  which  is  optimal 
considering  all  requirements.  Furthermore,  most  design  problems  can 
be  decomposed  into  subproblems  that  can  be  solved  independently. 
Doing  so,  however,  ignores  the  interaction  between  the  parts  and  often 
leads  to  suboptimal  solutions.  These  characteristics  of  the  domain 
provide  initial  points  of  contrast  between  experts  and  novices.  In  the 
next  section,  the  set  of  standard  procedures  for  designing  one  type  of 
simple  circuit,  combinational  logic  circuits,  is  described. 

Solving  Combinational  Logic  Circuit  Design:  Standard  Procedures 

There  is  a  standard  framework  for  solving  combinational  logic 
circuit  design  problems  (Table  2)  that  is  taught  in  the  introductory-level 
digital  circuit  design  course  at  the  imiversity  where  this  study  was 
conducted.  The  standard  framework,  presented  in  the  text  (Garrod  & 
Boms,  1991)  as  well  as  in  class,  includes  six  sequentially  ordered 
components  as  given  in  Table  2:  Understand,  Truth  Table,  Karnaugh 
Maps  (K-maps),  Boolean,  Implement,  and  Evaluate.  During  the 
Understand  Component,  the  designer  is  reading  the  problem,  trying  to 
figure  out  what  type  of  circuit  needs  to  be  designed,  deciding  how  to 
design  the  circuit,  what  the  inputs  and  outputs  are  and  how  to 
represent  them.  The  first  step  of  the  actual  design  is  the  Truth  Table 
Component.  In  this  crucial  step,  a  tmth  table  is  generated  which  is  a 
formalization  of  the  problem  statement,  describing  the  input-output 
relationships  of  the  desired  circuit.  Then,  the  best  way  to  simplify  the 
logic  expressions  derived  from  the  truth  table  is  to  carry  out  the  K-map 
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Component.  A  K-map  is  an  intermediate  step  to  the  formulation  of 
logical  expressions.  Although  this  step  can  be  omitted,  it  allows  the 
design  to  visually  check  for  certain  patterns.  These  patterns  help 
designers  identify  overlapping  functionalities  and  patterns  in  the  truth 
table,  K-map,  and  Boolean  expressions  that  have  standard 
implementation  layouts. 


Insert  Table  2  about  here 


Next,  the  information  from  the  K-map  is  used  to  generate 
Boolean  expressions  in  the  Boolean  Component.  Boolean  algebra  is  a 
systematic  algebraic  formulation  of  mathematical  postulates  and 
symbols  that  can  be  used  to  describe  and  simplify  logic  functions. 
Boolean  relationships  can  be  used  to  substitute  one  type  of  gate  for 
another,  which  is  a  useful  technique  to  optimize  the  construction  of 
digital  logic  circuitry.  During  the  Implement  Component,  the  Boolean 
equations  are  represented  by  basic  logic  hmctions,  each  of  which 
corresponds  to  a  logic  gate.  Logic  gates  are  the  fundamental  building 
blocks  of  digital  electronics.  After  implementing  the  function,  the 
designer  refers  back  to  the  original  desired  ftmctional  description  and 
evaluates  whether  the  circuit  solution  accomplishes  the  functionality 
{Evaluate  Component). 

The  standard  procedure  t5rpically  applies  to  single  output  circuits 
and  the  optimal  solution  involves  sequential  execution  of  each 
component.  When  the  circuit  has  more  than  one  output,  however,  as 
the  ones  used  in  this  experiment,  designers  have  some  flexibility  in  the 
solution  process.  They  can  choose  to  solve  each  output  separately  or  to 
solve  them  simultaneously.  Solving  the  outputs  simultaneously  can 
lead  to  finding  more  ways  to  optimize  the  solution,  e.g.,  sharing  gates 
between  the  two  outputs  to  reduce  duplication  of  gates  between  the 
parts  of  the  circuit.  Designers  may  choose  to  switch  back  and  forth 
between  the  two  outputs,  e.g.,  designers  may  solve  the  K-map  for 
output  1  and  then  solve  the  K-map  and  the  Boolean  equations,  and 
implement  output  2  before  solving  the  Boolean  equations  for  output  1. 
The  designer  has  a  number  of  options  regarding  the  order  in  which 
parts  of  the  solution  are  tackled. 

In  the  present  research,  a  combinational  logic  circuit  design 
problem,  a  one-bit  subtracter,  was  given  to  students  enrolled  in  the 
first  course  in  circuit  design.  The  course  instructor  provided  an  expert 
solution:  sequential  execution  of  components  with  several  attempts 
made  to  minimize  the  number  of  gates.  Two  experts  who  were  not 
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academic  instructors  also  solved  the  subtracter  problem^.  Hence,  the 
study  compared  the  performance  of  experts  to  novice  students,  using 
the  class  instructor's  performance  as  the  benchmark. 

Expected  Differences 

We  expected  that  the  more  expert  subjects  would  exhibit  more 
planning  behavior  than  the  less  expert  subjects.  In  the  beginning  of 
their  design  process  we  expected  the  more  expert  subjects  to  construct 
global  features  of  the  design  process  before  they  began  (e.g.,  Larkin, 
1980).  Based  on  the  literature  that  describes  experts  as  having  strong 
self-monitoring  skills  (e.g.,  Chi,  1978;  Chi,  1987;  Chi,  Glaser,  &  Rees, 
1982;  Larkin,  1983;  Miyake  &  Norman,  1979;  and  Simon  &  Simon, 
1978),  we  expected  that  the  more  expert  subjects  would  verify  and 
evaluate  each  step  of  the  solution  and  check  that  their  final  solution 
did  indeed  fimction  properly.  Based  on  Akin's  (1988)  discussion  of 
expert  architects'  problem  restructuring  and  creation  of  alternative 
scenarios  we  expected  the  more  expert  designers  to  choose  among 
alternative  solutions  in  attempting  to  achieve  an  optimal  solution. 
We  expected  that  the  less  skilled  subjects  would  exhibit  little  planning 
in  the  beginning  of  their  solution  process  but  would  have  to  stop  later 
to  decide  how  to  proceed  when  at  an  impasse  (e.g.,  Paige  &  Simon, 
1966). 

Method 


Participants 

The  novices  were  five  undergraduate  student  volimteers  (one 
female  and  four  males)  who  participated  in  the  study  (RP,  JB,  DT,  ES, 
and  TS).  All  five  students  were  enrolled  in  a  senior-level  digital  design 
course.  The  students  had  all  completed  one  introductory  course  in 
digital  circuits  and  had  very  limited  design  experience.  All  students 
had  been  taught  the  standard  procedure  for  dealing  with  a  problem  of 
this  sort.  The  students  were  paid  by  the  hour  for  their  participation. 

The  experts  (EM  and  AK)  were  two  advanced  graduate  students 
each  with  a  master's  degree  in  electrical  engineering.  One  expert  had 


^Often  in  expertise  research,  expert  subjects  are  academic  instructors  who  are 
accustomed  to  teaching  problem-solving  and  explaining  how  to  do  it.  In  the  present 
research  we  recruited  experts  who  were  not  accustomed  to  talking  about  how  to  solve 
these  problems. 
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over  five  years  of  professional  experience  as  well.  The  other  expert  did 
not  have  professional  experience  but  had  completed  several  advanced 
courses  in  digital  circuits  and  a  related  project. 

The  instructor  of  the  senior-level  digital  design  course  in  which 
all  of  the  imdergraduate  student  subjects  were  enrolled  provided  us 
with  an  independent  rating  of  the  skill  level  of  each  student  subject 
based  on  class  performance  at  the  end  of  the  semester.  Each  subject  was 
rated  on  a  scale  from  1  to  10,  with  1  reflecting  little  skill  and  10 
indicating  high  skill.  Table  3  summarizes  the  instructor's  skill  ratings 
of  each  student  subject. 


Insert  Table  3  about  here 


Design  Task  and  Procedure 

Subjects  were  given  the  following  design  task: 

You  are  to  design  a  1-bit  full  subtracter.  Your  circuit  will 
accept  one  bit  of  each  operand  and  an  input  borrow  bit  and 
produce  a  result  bit  and  an  v  itput  borrow. 

In  solving  this  problem  keep  in  mind  that  we  are  not 
interested  in  how  much  time  it  will  take  you.  We  are  interested 
in  seeing  a  good  design  and  the  steps  leading  you  there.  Assume 
you  will  have  to  make  a  product  out  of  this  design.  The  circuit 
will  have  to  be  implemented  in  a  CMOS  chip.  Your  criteria  for 
"good  design"  should  be  (in  order  of  importance): 

1.  Minimum  cost,  that  is,  minimum  number  of  gates  used.  Keep 
in  mind  that  some  CMOS  gates  are  more  complicated  than 
others.  The  goal  is  thus  actually  to  minimize  the  area  of  the 
chip's  layout. 

2.  Testability 

The  problem  required  the  subjects  to  design  a  one  bit  full  subtracter, 
taking  into  account  several  design  constraints.  The  problem  was 
similar  to  an  example  often  used  in  class  (an  adder  circuit),  but 
different  enough  so  that  neither  novices  nor  experts  could  generate  the 
specific  solution  for  this  problem  from  memory.  All  subjects  had  been 
exposed  to  the  backgroimd  knowledge  needed  for  solving  this  type  of 
problem.  The  circuit  required  three  inputs  and  two  outputs,  and  the 
specified  target  representation  was  a  circuit  at  the  gate  level.  This 
problem  may  be  solved  algorithmically  with  the  designer  proceeding 
through  the  components  of  combinational  logic  circuit  design  in  the 
order  in  which  they  were  previously  described.  As  discussed 
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previously,  the  designer  may  solve  for  each  of  the  outputs 
simultaneously  or  separately. 

The  subjects  were  provided  with  the  foregoing  written, 
functional  description  of  the  problem.  They  were  asked  to  think-aloud 
during  the  solution  process  and  the  sessions  were  videotaped.  After 
the  specific  instructions  about  thinking  aloud,  each  subject  was  given  a 
practice  problem  to  help  them  get  used  to  thinking  aloud  and  to  the 
presence  of  a  video  camera.  Each  subject  then  solved  the  problem 
independently  and  without  a  time  constraint,  using  paper  and  dark 
markers  that  we  supplied.  Immediately  following  the  design  session, 
each  subject  was  asked  to  watch  the  tape  of  her  or  his  session  and  to 
make  additional  comments  about  what  they  were  doing  or  thinking 
and  why.  This  retrospective  session  was  also  videotaped. 

Analysis  and  Coding  of  Protocols 

The  solutions  were  evaluated  with  respect  to  correctness  and 
optimization  of  the  solution.  Also,  the  protocol  videotapes  for  the 
problem  were  coded  for  standard  components  of  solution  and  discrete 
behavioral  episodes.  The  component  analysis  determined  whether  the 
subject  had  used  the  previously  described  standard  sequence  of 
solution  components  for  doing  combinational  logic  circuit  design.  We 
added  an  Other  Component  to  encompass  any  components  used  by  the 
subjects  in  solving  the  problem  that  were  different  from  the  standard 
components. 

All  coding  was  done  using  a  typed  transcript  of  the  audio  portion 
of  each  session,  the  written  notes  produced  by  the  subject,  and  the 
videotape  of  the  problem-solving  session.  In  addition  to  the 
component  analysis,  the  solutions  were  analyzed  into  discrete 
behavioral  episodes  using  a  set  of  i2  episode  types  which  characterize 
the  nature  of  the  activity  the  subject  was  performing  at  a  particular 
point  in  the  solution  (cf.  Ullman,  Dietterich,  &  Stauffer,  1988).  The 
episode  traces  were  then  validated  against  the  subject's  retrospective 
comments  (audio  transcripts  and  videotapes)  for  consistency.  This  was 
done  to  ensure  that  the  coders  accurately  categorized  the  episodes;  this 
was  especially  important  for  the  planning  and  evaluation  episodes. 

Episodes  characterize  the  nature  of  the  activity  being  performed 
at  a  particular  point  in  the  solution.  The  full  set  of  12  episodes  are 
listed  in  the  Appendix,  together  with  a  brief  description.  Here  we 
discuss  the  five  types  of  design  episodes,  that  are  the  focus  of  this  paper. 
These  are  the  five  episodes  associated  with  planning-related  activity: 
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plan,  evaluate,  verify,  list  and  select.  Plan  episodes  contain 
information  about  activities  that  will  be  carried  out  in  solving  the 
problem  or  a  portion  of  the  problem.  The  following  italicized 
comment  is  an  example  of  a  plan  episode:  DT  implements  the  Borrow- 
Chit  output  and  then  says,  "Okay  that  should  be  our  borrow-out  bit,  and 
I  will  come  back  to  double  check  that  in  a  minute.  "  Evaluate  episodes 
assess  the  progress  of  a  plan  with  respect  to  the  current  state  of  the 
solution  and  the  validity  of  the  active  goal  structure.  For  example, 
after  EM  had  specified  ^e  truth  table,  he  looked  at  his  work  and 
evaluated  saying:  "That's  not  correct,  this  is  wrong"  (pointing  to 
Borrow-Out  in  the  truth  table). 

Verify  episodes  compare  the  current  state  of  the  design  with  a 
previous  state  or  with  the  design  requirements.  Looking  for  errors  in 
the  transition  from  one  component  to  the  next  is  one  example  of  a 
verify  episode,  as  illustrated  by  JB  checking  the  circuit  implementation 
against  the  truth  table:  "And  I  am  just  going  to  finally  check  exactly 
what  it  was  I  was  designing  for." 

List  episodes  organize  information  into  competing  paths  for 
solving  the  problem  or  subproblem.  An  example  of  a  list  episode  is 
given  in  Table  4.  DT  was  pursuing  ways  to  combine  the 
implementations  of  the  Result  and  Borrow-Out  outputs  because  he 
realized  his  current  implementation  for  Borrow-Out  may  not  have 
been  minimal  (segment  1).  His  first  implementation  was  a  three-input 
implementation.  DT  verbalized  an  alternative  way  to  minimize  the 
design  using  a  two-input  implementation  (segments  2  through  6).  In 
segment  7,  he  made  a  comparison  between  the  two  input  and  the 
three  input  implementations. 


Insert  Table  4  about  here 


Select  episodes  tend  to  follow  list  episodes  because  they  compare 
and  choose  among  the  competing  solutions.  For  example,  EM 
simulated  the  circuit  to  count  gates  and  selected  an  option  that  was  not 
a  combined  solution  becau.se  it  was  less  expensive  in  terms  of  gates: 

"So  this  is  actually  one  input  more  expensive  so  the  best  thing  to  do  is 
to  uh,  is  the  first  solution." 

The  difference  between  list  and  select  episodes  compared  to  plan 
episodes  is  that  list  and  select  episodes  are  restricted  to  enumerating 
and  deciding  among  alternative  actions  whereas  in  plan  episodes  the 
designer  is  laying  out  a  solution  strategy.  When  the  subject  tries  to 
decide  between  a  NAND  and  a  NOR  implementation,  that  is  an 
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example  of  list  and  select  episodes.  When  he's  deciding  what  to  do 
next,  either  to  follow  the  truth  table  with  a  K-map  or  Boolean 
equations,  is  a  "ian  episode. 

Reliability  oetween  two  coders  was  measured  with  regard  to 
three  aspects  of  episode  coding:  (a)  identification  and  parsing  of  the 
solution  into  episodes,  (b)  the  type  of  episode,  and  (c)  descriptive 
comments  about  the  problem  solving  function  or  the  input /output 
component  involved  in  the  episode.  Two  experimenters  were  first 
trained  to  recognize  the  episodes  in  the  protocols.  They  then  each 
generated  an  episode  trace  for  a  problem  and  these  traces  were  used  for 
computing  reliability  on  all  three  measures.  Agreement  of  over  85% 
was  established  on  all  three  aspects.  Disagreements  were  resolved  in 
discussion. 

Errors  were  judged  relative  to  the  instructor’s  solution.  The 
errors  in  each  solution  were  coded  such  that  errors  did  not  propagate 
across  components.  Because  of  the  sequential  nature  of  the  problem, 
an  error  that  occurred  early  on  would  have  repercussions  in  all 
subsequent  steps.  Each  component  was  scored  assuming  that  the  input 
from  the  previous  component  was  correct.  This  was  done  to  get  an 
accurate  count  of  the  t^e  and  location  of  errors.  Errors  were  also 
tabulated  for  each  subject  and  the  component  in  which  they  occurred 
was  identified. 

Results  and  Discussion 

An  initial  examination  of  the  episode  coding  and  error  analyses 
revealed  two  surprising  findings.  First,  the  frequency  and  location  of 
plan  episodes  was  not  consistent  with  what  we  had  expected  based  on 
previous  research.  There  was  only  a  single  case  in  which  any  subject 
laid  out  a  solution  plan  for  the  entire  problem  at  the  outset  of  problem 
solving.  In  the  vast  majority  of  plan  episodes,  planning  and  re¬ 
planning  were  going  on  throughout  the  solutions,  and  were  just  as 
frequent  in  the  students'  protocols  as  in  the  experts'.  Second,  none  of 
the  subjects  solved  the  problem  correctly,  including  the  experts.  In  fact, 
the  nature  of  the  errors  was  such  that  they  substantially  changed  the 
nature  of  the  circuit  and  may  have  necessitated  the  extensive  planning 
and  replanning  that  we  observed. 

In  reporting  our  findings,  we  first  consider  the  overall  solution 
plan  produced  by  one  of  the  experts.  We  then  consider  the  general 
nature  and  accuracy  of  the  problem  solving  as  a  context  wittiin  which 
to  examine  interspersed  planning,  replanning  and  the  emergence  of 
the  concept  of  Planning-Related  Activity.  We  then  focus  on  Planning- 
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Related  Activity  conditional  on  errors. 

An  Expert's  Overall  Solution  Plan 

One  of  the  experts,  EM,  provided  an  overall  plan  for  his  solution 
prior  to  actually  beginning  to  work  on  the  truth  table.  The  transcript  of 
the  plan  is  provided  in  Table  5.  The  first  step  is  to  identify  the  input 
and  output  bits  (segment  1).  The  next  goal  EM  describes  is  finding  a 
function  (segment  2).  He  indicates  how  he  will  do  that  using  a  K-map 
(segment  3)  and  provides  the  procedural  details  of  K-map  construction 
in  segment  4,  ending  the  segment  by  stating  his  goal  of  finding  the 
prime  implicants  for  the  fimction.  In  segment  5,  EM  indicates  that  he  is 
trying  to  satisfy  the  minimization  constraint  on  circuit  design  and  in 
segment  6,  he  indicates  that  he  will  try  and  combine  the  two  parts  of 
the  circuit  in  order  to  do  so.  He  indicates  some  uncertainty  about 
whether  this  will  work  (segment  7).  In  the  final  two  segments,  EM 
discusses  implementation  of  the  function  and  considers  alternative 
gate  types  that  might  be  used,  commenting  that  "from  there  it's  very 
simple." 


Insert  Table  5  about  here 


It  is  clear  from  the  outset  that  EM  had  a  series  of  goals  in  mind 
and  that  these  included  optimizing  the  solution.  It  is  interesting  to 
note  that  EM  ran  into  trouble  in  designing  this  circuit  when  he  reached 
the  implementation  phase,  the  part  of  the  solution  that  he  indicated 
would  be  simple. 

General  Nature  of  the  Solutions 

Just  as  we  had  expected  the  experts  to  be  more  likely  to  produce  a 
global  plan,  we  had  expected  that  they  would  solve  the  circuit  design 
problems  faster  than  students.  This  turned  out  not  to  be  the  case,  either 
in  terms  of  total  time  spent  solving  the  problem  or  in  terms  of  the 
number  of  episodes.  Subjects  took  from  14  minutes  to  94  minutes  with 
no  clear  trend  across  expertise  level  evident.  These  data  are  shown  in 
the  upper  portion  of  Table  6.  The  individual  subjects  are  grouped  in 
terms  of  whether  or  not  they  made  an  error  in  the  truth  table  and  then 
ordered  in  terms  of  their  instructor  ratings  with  "ex"  indicated  for  the 
expert  subjects.  No  general  trend  is  discernible  in  terms  of  total  number 
of  episodes,  in  part  because  of  the  degree  to  which  the  individuals  tried 
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to  deal  with  the  optimization  constraints  and  in  part  due  to  the  kinds 
of  errors  they  made  in  the  truth  table. 


Insert  Table  6  about  here 


By  considering  the  amount  of  time  spent  in  the  Understand 
Component,  we  can  better  determine  what  the  subjects  who  did  not 
formulate  a  global  plan  did  before  they  began  the  problem.  AK,  the 
other  expert,  spent  just  over  5  minutes  in  the  Understand  Component, 
while  among  the  student  subjects,  RP  spent  just  over  1  minute,  JB 
spent  just  under  1  minute,  DT  spent  about  1  minute,  ES  spent  just  over 
2  minutes,  and  TS  spent  just  over  1  minute  in  the  Understand 
Component.  It  is  evident  that  the  student  subjects  did  little  more  than 
read  the  problem  statement  before  they  began  the  truth  table.  These 
times  can  be  contrasted  with  EM’s,  the  expert  who  did  formulate  a 
global  plan.  EM  spent  8  minutes  in  the  Understand  Component 
detailing  the  steps  of  his  solution,  as  previously  discussed.  So  although 
AK  did  not  given  an  overt  plan  the  time  he  spent  compared  to 
students  may  indicate  some  time  of  planning  at  the  onset. 

Comparisons  of  the  nature  of  the  solution  activity  following  the 
Understand  Component  indicate  that  simple  predictions  of  total  time 
and  fewer  episodes  oversimplify  the  nature  of  the  design  process.  The 
two  experts  differed  by  34  episodes  and  40  minutes.  The  shorter 
solution  was  EM's.  He  started  with  a  global  plan.  Although  AK's 
failure  to  start  with  a  global  plan  may  have  contributed  to  his  having  a 
longer  solution  process  than  EM,  an  equally  plausible  explanation  is 
that  AK  devoted  substantial  amoimts  of  time  to  considering  two 
possible  implementations.  TTie  following  extract  from  his  protocol 
illustrates  the  implementations  he  considered,  including  his  action  of 
counting  gates  to  determine  which  implementation  would  give  him  a 
more  optimal  solution. 

1.  Okay,  so  now  let's  compare  two  solutions 

2.  Solution  one  for  B2  is  (draws  label:  Solution  1  for  B2) 

3.  12,  Bl.  Then  the  gate  is  this.  II  and  I  take  the  gate  here 
directly  and  I  need  to  have  a  NAND  gate  here,  (draws 
implementation  for  Solution  1  for  B2)  B2.  This  is  the  first 
solution,  (indicates  the  circuit  he  has  just  drawn) 

4.  Then  solution  two  is  for  B2  (draws  label:  Solution  2  for  B2) 
(it)  is  a  three  gates  solution,  one  is  II,  12,  Bl. 

5.  The  other  is  I  guess,  the  same  thing  for  this,  oh  it  is  not.  This 
is  3  input  NAI^  gate:  II,  12,  Bl  and  one  inverter,  NAND 
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gate,  direct  NAND  gate  and  NAND  gate,  (draws 
implementation  for  Solution  2  for  B2) 

6.  B2  and  I  have  oh,  no  no  no  (indicates  an  error  in  Solution  2) 
This  is  wrong,  this  is  direct,  (crosses  out  a  NAND  gate) 

I  need  one  inverter  here,  from  II  (adds  an  inverter  gate  to 
Solution  2)  and  the  other  way.  I’m  sorry  is  direct,  (adds  an 
inverter  gate  to  Solution  2) 

7.  1,2,3,4,5  (counts  gates  in  solution  1) 

8.  1,2,3,4,5  (counts  gates  in  solution  2) 

9.  inverter  is  two,  (points  to  inverter  in  Solution  2) 

10.  inverter  is  three  (points  to  inverter  in  Solution  1) 

11.  There  is  almost  no  improvement.  So  let's  take  this,  (makes 
a  check  mark  by  Solution  1  to  indicate  his  choice) 

This  iterative  process  of  choosing  among  alternative  implementations 
to  find  the  best  possible  solution  accotmts,  in  part,  for  the  greater 
number  of  episodes  for  AK  compared  to  EM.  EM  considered 
alternatives  earlier  on  in  the  Boolean  Component.  Working  with  the 
symbolic  representations  in  these  equations  occupied  less  time  and 
fewer  episodes. 

A  second  reason  AK  had  a  greater  number  of  episodes  than  EM 
was  due  to  the  fact  that  AK  considered  cost.  AK  was  not  primarily 
concerned  with  the  mechanics  of  how  to  solve  the  problem  but  rather 
with  the  best  design  to  meet  the  functional  requirements  in  the 
problem  statement.  He  was  concerned  not  only  with  minimizing  the 
number  of  gates  but  also  with  minimizing  the  cost  in  building  the 
circuit.  No  other  subject  asked  questions  concerning  cost.  The 
following  is  an  excerpt  from  a  dialogue  between  AK  and  the 
experimenter  (in  bold)  concerning  cost  and  minimization: 

AK:  Okay  now  in  order  for  the  cost  to  be  minimum.  I've  a 
question:  Do  you  assign  any  costs  to  each  type  of  gate?  Like  AND 
is  some  cost,  NAND  is  some  cost,  NOR  is  some  cost,  NOT  is 
some  cost;  Or  can  I  just  assume  that? 

E:  Yes. 

AK:  Oh,  yes?  Okay,  so  how  much  AND  gate  costs  more  than  a 
NAND  gate? 

E:  Um ...  the  order...  Let  me  put  it  to  you  this  way:  NOT, 

NAND2,  NOR2,  NAND3,  NOR3,  etc.  This  is  the  order  of 
increasing  cost,  okay? 

AK:  So  first  let's  think  about  the  cost  of  minimum  solution.  Uh 
okay,  hmm,  what  is  the  smallest  logical  representation  of  this? 
We  think  that  AK's  concern  over  the  cost /minimization  issues  reflects 
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his  experiential  base  in  the  commercial  world.  The  other  expert  did  not 
have  ^is  kind  of  experience  and  did  not  consider  cost. 

Examination  of  the  reasons  for  the  large  number  of  episodes 
among  the  student  subjects  is  also  revealing  of  the  nature  of  their 
solution  processes.  Compared  to  the  experts  and  to  the  other  students, 
DT  and  ES  had  a  large  number  of  episodes.  The  reasons  for  these  are 
different  in  the  two  cases.  Although  DT  had  correctly  completed  the 
truth  table,  he  used  a  nonstandard  component  in  his  solution.  After 
completing  the  truth  table,  instead  of  creating  a  K-map,  DT  decided  to 
look  "by-hand"  for  the  patterns  that  would  help  minimize  the  design. 
The  following  statement  by  DT  illustrates  the  beginning  of  the 
nonstandard  component: 

Now,  what  we  need  to  do  is  begin  figuring  out  what  conditions 
we've  got  for  each,  so  for  our  borrow  bit,  the  patterns  that  will 
set  it  are,... 

After  DT  has  looked  for  the  patterns  that  will  help  to  minimize  the 
design  he  plans  his  next  step: 

And,  that  is  this.  Okay  these  are  the  four  conditions  where  we 
have  got  a  one  on  the  output.  Now  I  am  looking  for  similarities 
at  this  point,  and  not  finding  too  many,  okay.  Either,  I  could  use 
a  Karnaugh  map,  or  just  try  and  work  it  at  gate  level  and  set  up 
the  gates  and  then  minimize. 

This  decision  occurred  early  in  the  solution  and  DT  tried  to  implement 
directly  from  the  truth  .table.  These  efforts  were  unsuccessful  but 
generated  18%  of  his  total  episodes.  He  then  decided  to  do  a  K-map  for 
one  of  the  outputs  and  tried  to  implement  again.  Thus,  DT's  large 
number  of  episodes  was  due,  in  part,  to  trying  to  take  a  shortcut;  the 
shortcut  failed. 

On  the  other  hand,  the  large  number  of  episodes  for  ES  were 
attributed  to  the  fact  that  he  was  not  able  to  do  the  truth  table  correctly, 
and  retried  this  many  times.  In  fact,  over  70%  of  his  episodes  were  part 
of  the  Truth  Table  Component.  ES  made  seven  attempts  at  specifying 
the  truth  table  before  he  foimd  one  that  he  accepted  and  used  to  design 
the  circuit.  Some  of  his  activities  during  this  time  were  rereading  the 
problem  statement  twice,  looking  in  books  for  ideas  ten  different  times, 
and  asking  the  experimenters  for  help.  ES  aptly  described  his  own 
situation: 

Still  looking,  still  thinking,  still  not  getting  very  far.  Whatever 
design  I  come  up  with  will  most  certainly  not  be  the  minimum 
design.  Because  you  have  to  have  a  very  good  grasp  of  what  you 
are  doing  to  be  able  to  get  a  small  design. 
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ES  received  the  lowest  rating  of  any  of  the  subjects  from  the  course 
instructor.  The  extent  of  this  lack  of  xmderstanding  is  manifest  in  the 
following  comment  by  ES:  "So  I  guess  to  minimize  to  design  the 
things  you  are  supposed  to  use  is  a  Karnaugh  map.  A  little  four  thing 
but  it's  only  a..."  It  was  obvious  that  he  knew  what  the  standard 
procedures  were  but  not  how  or  when  to  do  them. 

Among  the  remaining  students,  JB  got  the  truth  table  correct  and 
solved  the  problem  quickly,  in  14.3  minutes.  JB  used  the  standard 
components  but  omitted  the  K-map  step,  and  had  one  of  the  lowest 
number  of  episodes.  She  made  an  error  in  the  Boolean  Component, 
however,  and  did  not  minimize  the  number  of  gates  appropriately. 

Similarly,  RP's  solution  was  quick  but  inaccurate,  with  an  error 
in  the  truth  table  leading  to  a  circuit  that  did  not  map  onto  the 
functional  specifications  stated  in  the  problem.  RP  did,  however,  use 
the  standard  sequence  of  components  to  generate  his  solution. 

Finally,  TS  generated  a  moderately  lengthy  solution  (30 
minutes).  Although  TS  made  nine  errors  in  the  truth  table,  he 
followed  the  standard  sequence  of  components  and  did  attempt  to 
optimize  by  having  the  Result  and  Borrow-Out  solutions  converge 
(share  gates).  However,  TS  attempted  to  minimize  in  two  separate 
components.  He  spent  32.1%  of  his  total  episodes  in  the  Boolean 
Component  rewriting  the  Boolean  Equations  and  counting  gates  in  an 
effort  to  minimize  his  solution.  Then  TS  tried  separate 
implementations  producing  21.4%  of  his  total  episodes  during  the 
Implement  Component.  These  efforts  to  minimize  the  number  of 
gates  may  have  led  to  the  somewhat  larger  number  of  episodes. 

In  summary,  the  solutions  of  the  experts  and  the  student 
subjects  differed  with  respect  to  time  and  number  of  episodes  but  the 
experts  generally  took  longer  than  the  students,  contrary  to  our 
expectations.  The  analysis  of  aspects  of  the  design  process  generating 
the  lengthier  solution  procedures  reflected  the  kinds  of  distinctions 
among  experts  and  novices  that  have  been  noted  in  the  literature,  that 
the  experts  represent  a  problem  in  their  domain  at  a  deeper  level  and 
do  more  upfront  planning  than  the  novices.  However,  we  also  found 
evidence  of  differences  between  the  approaches  taken  by  our  two 
experts.  How  designers  attempted  to  optimize  their  solutions  was  a 
major  source  of  differences  among  them. 

Solution  Errors 

None  of  the  subjects  produced  a  correct  solution.  We  had  not 
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anticipated  that  the  experts  would  have  any  trouble  with  this  problem 
but  they  reported  that  in  their  everyday  activities  they  no  longer  have 
to  design  at  the  gate  level.  While  they  once  knew  the  process,  it  was  no 
longer  at  their  fingertips.  Across  all  of  the  subjects,  the  locus  of  errors 
varied  across  components  but  the  highest  percentages  of  errors  were  in 
the  truth  table  component. 

Errors  in  the  Truth  Table  Component.  The  most  important  step 
in  the  design  is  going  from  the  English  language  description  of  the 
circuit  functionality  to  the  truth  table.  An  inaccurate  truth  table  usually 
means  that  the  circuit  being  designed  is  different  from  the  intended 
one,  which  may  significantly  alter  the  nature  of  the  problem,  making 
pattern  recognition  more  difficult  (or  sometimes  easier)  than  would 
have  otherwise  been  the  case.  Moreover,  in  combinational  logic  circuit 
design,  designers  often  verify  final  designs  against  truth  tables.  In  this 
manner,  they  verify  that  a  final  design  agrees  with  its  truth  table. 
However,  if  the  truth  table  is  not  correct,  the  circuit  will  not  execute  the 
desired  functionality,  in  this  case  subtraction. 

Only  three  of  the  subjects  correctly  specified  the  truth  table:  one 
of  the  experts,  EM,  and  two  of  the  students,  JB  and  DT.  These  students 
were  two  of  the  three  who  received  the  highest  ratings  from  the  course 
instructor  (Table  3).  The  second  expert,  AK,  made  a  careless  error  in  the 
truth  table,  as  did  the  three  remaiiung  students.  It  is  interesting  that, 
despite  the  error  that  AK  made  in  the  truth  table,  he  correctly 
implemented  the  circuit  he  specified  in  the  truth  table.  The  students 
did  not  correctly  implement  the  circuits  specified  by  their  truth  tables, 
making  additional  errors  in  other  components.  This  is  not  surprising 
because  the  errors  in  the  truth  table  prevented  students  from  seeing 
patterns  for  standard  gates.  They  were  attempting  to  solve  more 
complicated  circuits  than  the  one-bit  subtracter. 

Errors  in  Other  Components.  Other  than  errors  in  the  truth 
table,  the  component  with  the  greatest  number  of  errors  was 
Implement.  EM  and  DT  both  made  errors  implementing  the  Result 
and  the  Borrow-Out  expressions,  as  did  the  three  students  who  got  the 
truth  table  incorrect.  EM  also  made  a  conceptual  error  in  grouping  the 
K-maps. 

The  faulty  solutions  and  the  virtual  absence  of  global  planning 
provide  some  explanation  for  the  imexpected  finding  that  we  alluded 
to  earlier:  There  were  more  instances  of  plan  episodes  than  we  had 
anticipated,  especially  among  the  student  subjects,  and  planning  and 
replanning  episodes  occurred  throughout  the  solutions.  In  the  next 
section  we  consider  activities  related  to  planning. 
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Planning-Related  Activity 

As  indicated  earlier,  EM's  overall  plan  was  the  only  incidence  of 
a  plan  episode  that  laid  out  an  entire  solution  strategy.  All  other 
instances  of  plan  episodes  occurred  in  more  restricted  circumstances 
and  were  limited  to  a  single  component  or  to  considering  how  to  go 
from  one  component  to  the  next.  These  plan  episodes  were  local  in 
scope  and  tended  to  co-occur  regularly  with  four  other  episodes:  verify, 
evaluate,  list  and  select  (see  the  Appendix  for  definitions).  There  was  a 
functional  integrity  to  these  groupings  of  episodes  that  led  us  to  call 
them  Planning-Related  Activity.  This  activity  often  occurred  when  the 
subject's  verification  or  evaluation  episodes  were  negative;  in 
response,  the  subject  would  consider  other  ways  to  approach  the 
problem  and  list  alternatives  before  selecting  among  them.  In  contrast, 
when  the  subject's  verification  or  evaluation  episodes  were  positive, 
the  subject  would  plan  the  next  step  in  the  solution. 

An  excerpt  from  AK's  protocol  illustrates  part  of  the  relationship 
among  the  episodes  that  constitute  Planning-Related  Activity  (see 
Table  7).  AK  had  just  finished  specifying  the  truth  table.  In  segment  1 
he  planned  to  verify  the  truth  table.  In  segment  2  he  verified  the  truth 
table  by  tracing  his  pen  over  each  line  of  the  truth  table  while  checking 
his  subtraction.  In  the  process  of  verifying  the  truth  table  he  evaluated 
that  line  3  had  an  error  in  the  output  (segment  3).  In  segment  4,  AK 
patched  the  error.  Following  this  he  continued  to  verify  (segment  5). 
Once  again  while  verifying,  he  evaluated  that  there  was  an  error,  this 
time  in  line  8  (the  last  line  of  the  truth  table;  see  segment  6).  In 
segment  7,  AK  patched  the  error.  Finally ,  in  segment  8,  he  plarmed 
the  next  step,  to  work  at  the  gate  level.  In  this  portion  of  AK's  protocol 
the  functional  interrelationships  among  the  individual  episodes  that 
constitute  Planning-Related  Activity  are  clear.  A  similar  coherence 
among  the  five  types  of  episodes  that  constitute  plarming-related 
activity  was  found  in  the  protocols  of  the  other  six  subjects. 

Insert  Table  7  about  here 


Planning-Related  Activity  episodes  accoimted  for  about  30%  of 
the  total  number  of  episodes  for  the  two  experts  and  for  between  30% 
and  55%  of  the  episodes  of  the  students.  The  exact  figures  are  provided 
in  Table  6.  One  expert,  EM  had  a  lower  percentage  of  planning  related 
episodes  than  the  student  subjects,  probably  because  he  laid  out  his 
plan  in  the  beginning  and  did  not  need  to  do  much  planning 
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afterwards.  For  AK,  the  other  expert,  there  are  two  possible 
explanations  for  his  lower  percentage  of  planning-related  activity. 

First,  he  was  carrying  out  a  standard  set  of  solution  procedures  that 
may  have  at  one  time  been  automated.  The  major  work  AK  did  went 
into  reinstating  those  procedures  but  executing  them  once  reinstated 
was  relatively  automatic  and  did  not  stimulate  verbalizations.  Even 
though  AK  had  an  error  in  his  truth  table,  he  was  able  to  apply  the 
standard  procedures  and  continue  to  solve  the  problem.  Second,  AK  is 
a  native  Japanese  speaker  and  therefore  may  be  less  verbal  in 
expressing  his  ideas  in  English,  and  as  we  noted  earlier  the  time  he 
spent  in  the  Understand  Component  suggests  that  he  may  have  been 
planning  but  not  overtly  verbalizing. 

Among  the  students,  there  appears  to  be  a  relationship  between 
the  course-instructor's  rating  of  them  and  the  percentages  of  Planning- 
Related  Activity  that  they  demonstrated.  JB,  TS  and  DT  had  40%-55% 
Planning-Related  Episodes  (see  Table  6);  they  also  received  the  highest 
instructor  ratings.  The  students  with  the  lower  instructor  ratings  had 
relatively  low  percentages  of  Planning-Related  Activity  (29%-32%). 

The  student  data  appear  consistent  with  a  model  of  expertise  that 
predicts  more  planning  for  the  more  expert  student  subjects.  As  we 
indicated  above,  the  experts  had  a  similarly  low  percentage  of 
Plaiming-Related  Activity  but  for  different  reasons  than  the  less  expert 
student  subjects. 

Distribution  of  Planning-Related  Activity.  Planning-Related 
Activities  can  have  two  obvious  purposes:  (1)  determining  what  to  do 
after  a  component  or  phase  of  problem  solving  activity  has  been 
completed  and  (2)  determining  what  to  do  when  a  particular  problem 
solving  activity  seems  to  not  be  moving  the  solution  forward.  We  refer 
to  the  former  as  Across-Component  Plaiming-Related  Activity  and  the 
latter  as  Within-Component  Planning-Related  Activity. 

Across-Component  Planning-Related  Activity.  The  occurrence 
of  planning-related  activity  at  the  conclusion  of  a  particular  component 
suggests  that  subjects  were  evaluating  a  previous  step  against  the 
current  step  or  were  planning  the  next  step.  The  distribution  of  the 
Across-Component  Planning-Related  Activity  suggests  that  the  better 
student  subjects  did  more  Across-Component  planning  (Table  8).  The 
two  student  subjects  who  had  the  highest  percentage  of  Across- 
Component  Planning-Related  Activity  were  JB  and  TS.  JB  and  TS  also 
had  the  highest  instructor  ratings.  ITie  following  example  illustrates 
the  occurrence  of  a  local  plan  after  the  successful  conclusion  of  a 
component  of  solution.  JB  had  implemented  the  Boolean  expression 
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for  Result;  she  then  planned  how  she  would  get  the  Borrow-Out 
expression: 

And  to  break  it  down  I  would  have  to  look  at  some  intermediate 
logic  to  see  if  I  can  get  my  borrow-out...without  having  to 
redesign  the  entire  circuit,  making  an  entirely  new  circuit  and 
try  to  utilize  what  I  have. 


Insert  Table  8  about  here 


Verify  and  evaluate  were  also  quite  common  at  the  Across- 
Component  jimctures.  TS  had  finished  implementing  his  design.  He 
looked  over  at  the  problem  statement  and  evaluated  how  he  had  made 
sure  that  his  circuit  met  the  testability  requirement.  At  the  end  he 
accepted  his  circuit,  stating: 

Now  it  says  for  testability,  (glances  at  problem  statement) 
Testability's  straight  forward,  all  I  have  is  three,  three  points: 

A  B  or  W.  (circles  the  inputs  A,  B  and  W  on  his  design  and 
points  to  them) 

That's  straight  forward  how  to  test  it  and  my  outputs;  sub  and 
out,  (underlines  sub  and  out  on  his  design  to  indicate  them) 
it's  strai'iht  forward.  That's  it.  (accepts  his  design) 

There  wt-re  few  instances  of  Across-Coniponent  Planning- 
Related  Activity  among  the  less  expert  students  and  the  planning- 
related  activity  that  did  occur  was  limited. 

The  more  skilled  designers  had  higher  percentages  of  Across- 
Component  Planning-Related  Activity  because  they  were  either 
plarming  their  next  step  or  verifying  and  evaluating  the  previous  step. 

Within-Component  Planning' Related  Activity.  When 
Planning-Related  Activity  occurred  within  a  component  it  appeared 
that  subjects  were  either  confused  or  unsure  of  what  they  were  doing; 
they  stopped  and  planned  or  re-planned  what  their  next  step  ought  to 
be.  Accordingly,  we  expected  to  see  a  low  percentage  of  Within- 
Component  Planning-Related  Activity  among  the  experts  and  more 
highly  rated  students.  The  distribution  of  the  Within-Component 
Planning-Related  Activity  is  given  in  Table  8.  It  provides  only  partial 
support  for  the  prediction.  Consistent  with  the  prediction,  EM,  an 
expert,  had  the  lowest  percentage  of  Within-Component  Planning- 
Related  Activity:  Once  he  began  to  execute  a  component  the  process 
operated  straightforwardly.  The  other  expert,  AK,  had  a  relatively  high 
percentage  of  Within-Component  Planning-Related  Activity;  however, 
the  nature  of  these  episodes  indicated  that  his  planning-related  activity 
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was  not  due  to  confusion  or  errors.  Rather  he  was  evaluating  his 
progress  (50%  of  Within-Component  Planning-Related  Activity), 
listing  and  selecting  among  alternative  solutions  (a  total  of  10%  of  the 
Within-Component  Planning-Related  Activity)  and  then  verifying 
(15%  of  Within-Component  Planning-Related  Activity).  Although  list 
and  select  episodes  were  relatively  infrequent,  their  occurrence 
indicated  that  AK  was  choosing  among  alternative  solutions  for  the 
best  possible  solution.  AK's  comment,  "Okay,  so  now  let's  compare 
two  solutions"  clearly  indicates  this. 

The  students  who  received  the  lowest  rating  from  the  instructor, 
RP  and  ES,  had  relatively  high  percentages  of  Within-Component 
Planning-Related  Activity  (see  Table  8).  These  student  subjects  planned 
and  replanned  when  in  tihe  middle  of  a  component;  they  got  lost  and 
did  not  know  what  to  do.  This  is  reflected  in  the  fact  that  the  majority 
of  their  Within-Component  Planning-Related  Activity  were  evaluate 
and  plan  episodes  (50%  and  40%  for  evaluate  and  25%  and  44%  for 
plan,  respectively).  For  example,  after  working  on  the  truth  table  for  45 
minutes,  ES  said: 

Now  I'm  more  confused  than  when  I  started,  (holds  pen,  no 
action) 

5,6,7,8  (checking  the  number  on  previous  pages,  so  that  he  can 
number  the  next  page) 

uh,  still  trying  to  figure  out  how  to  do  it.  (writes  down  page 
number) 

It's  one  of  those  things,  I  guess  in  like  in  engineering  where  it’s 
like  one  thing  missing  and  you  figure  out  what  it  is  and  it  all 
falls  into  place  and  you  just  do  it.  (holds  pen,  no  action) 

DT  also  had  a  high  percentage  of  Within-Component  Planning- 
Related  Activity.  As  previously  discussed,  DT  tried  non-standard 
procedures  in  his  design.  His  Within-Component  Planning-Related 
Activity  is  almost  equally  divided  between  evaluate,  verify  and  plan 
episodes.  This  indicates  that  he  was  stopping  and  plaiming  in  the 
middle  of  a  component  and  spent  a  lot  of  effort  evaluating  and 
verifying  the  outcomes  of  those  plans. 

The  students  who  received  the  highest  rating  from  the 
instructor,  JB  and  TS,  had  moderately  low  percentages  of  Within- 
Component  Planning-Related  Activity.  Both  JB  and  TS  had  almost 
equal  amoimts  of  Within-Component  and  Across-Component 
Planning-Related  Activity.  However,  their  behavior  was  interesting  in 
that  33.3%  of  TS's,  and  41.7%  of  JB’s  Within-Component  Planning- 
Related  Activity  were  listing  and  selecting  among  alternative 
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solutions.  When  we  compare  them  with  the  other  student  subjects, 

DT  at  6%,  RP  at  0%,  and  ES  at  4.3%,  we  find  that  the  more  expert 
student  subjects  were  more  likely  to  consider  alternative  solutions. 

The  data  appear  consistent  with  a  model  of  expertise  that 
predicts  more  upfront  organization  by  more  expert  subjects.  The  more 
skilled  designers  had  higher  percentages  of  Across-Component 
Planning-Related  Activity  because  they  were  either  planning  their  next 
step  or  verifying  and  evaluating  the  previous  step.  The  more  skilled 
designers  had  higher  percentages  of  Within-Component  Planning- 
Related  Activity  when  they  were  choosing  among  alternative 
solutions.  Whereas  the  less  skilled  designers  had  higher  percentages  or 
Within-Component  Planning-Related  Activity  because  they  planned 
and  replanned  when  in  the  middle  of  a  component;  they  got  lost  and 
did  not  know  what  to  do. 

Summary  and  Conclusions 

Studies  based  in  other  domains  have  characterized  experts  as 
spending  more  time  or  effort  on  understanding  a  problem  before  they 
begin  to  solve  it  (e.g.,  Larkin,  1980).  Such  findings  received  support  in 
the  study  reported  here.  The  student  subjects  immediately  began  to 
work  through  the  components  of  the  problem  solution  as  they  learned 
in  class,  rather  than  first  constructing  a  deeper  imderstanding  of  what 
role  each  component  played  in  the  problem-solving  process.  One  of 
the  experts  verbalized  a  comprehensive  plan  before  he  began  to  solve 
the  problem.  The  other  expert,  AK,  may  not  have  verbalized  a  plan 
before  he  began  the  problem  because  he  recognized  there  was  standard 
procedure  and  implemented  it.  He  did,  however,  spend  a  relatively 
large  amount  of  time  in  the  Understand  Component  before  beginning 
to  solve  the  problem. 

Although  they  did  not  construct  an  overall  plan,  the  more 
skilled  student  subjects  (JB,  TS,  and  DT),  exhibited  some  traits  that  we 
believe  could  be  characteristic  of  advanced  designers.  Two  of  the  three 
more  highly  rated  student  subjects  (JB  and  DT)  were  able  to  correctly 
construct  a  truth  table  from  the  English  language  description  of  the 
problem  to  the  truth  table.  This  is  an  important  step  in  understanding 
the  fimctionality  of  the  circuit  and  how  it  relates  to  the  circuit  design. 
All  three  of  these  students  tried  to  minimize  the  circuit  design  through 
use  of  shared  gates.  This  demonstrates  the  ability  to  see  the  circuit  as 
an  interacting  imit  rather  than  as  two  separate  circuits  to  be  tied 
together  later.  JB  and  TS  also  had  more  Across-Component  Planning- 
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Related  Activity  than  the  other  student  subjects.  This  suggests  that  they 
did  more  planning  of  the  next  step  and  evaluating  the  previous  step 
with  respect  to  their  plan. 

It  is  interesting  to  note  that  although  the  problem  was  quite  close 
to  being  a  standard  "adder"  problem,  neither  of  the  experts  could 
readily  design  the  circuit  without  first  thinking  about  the  nature  of  a 
subtracter.  There  were  two  ways  in  which  the  experts  acted  as  we 
expected:  One  of  the  experts  elaborated  a  global  plan  for  the  design 
process  and  the  other  expert  searched  among  various  solution 
possibilities  for  an  optimal  solution.  One  of  the  interesting  aspects  of 
this  study  is  that  the  "experts"  had  more  difficulty  solving  the 
subtracter  problem  than  we  anticipated.  This  appeared  to  be  related  to 
the  fact  that  these  experts  did  not  use  gate  level  logic  in  their  current 
day  to  day  activities;  hence,  when  they  had  to  use  it,  they  were  forced  to 
look  it  up  or  reconstruct  it  from  memory. 

Characterizing  an  individual  as  an  expert  is  difficult  and  in- 
depth  thought  must  be  given  before  a  person  is  labeled  as  such.  For  an 
individual  to  exhibit  "expert"  behavior,  the  individual  may  need  to 
have  recent  experience  in  the  sub-domain  of  interest.  This  is 
important  because  experts  do  not  merely  have  a  superset  of  skills  as 
compared  to  students;  they  may  in  fact  lose  some  of  the  skills  necessary 
to  reach  this  "expert"  state.  Our  so-called  experts  complained  that  they 
have  not  been  designing  circuits  at  the  gate  level  for  many  years 
because  their  jobs  did  not  entail  gate  level  design.  Accordingly,  they 
indicated  they  needed  to  brush  up  on  their  knowledge  and  problem¬ 
solving  abilities  of  gate  level  digital  circuit  design.  One  of  our  experts, 
AK,  commented  that  in  industry  he  commonly  designed  on  a  larger 
scale  and  the  simple  circuits  such  as  adders  and  subtracters  could  be 
foimd  on  the  shelf;  he  did  not  have  to  take  the  time  to  make  them 
himself.  However,  these  individuals  were  chosen  as  experts  because  of 
their  standing  as  advanced  engineering  graduate  students  with 
experience  in  digital  circuit  design  and  because  they  were  not  academic 
teachers.  We  suggest  that  researchers  need  to  be  careful  whom  they 
label  as  an  "expert."  In  future  research,  experts  should  be  chosen  for 
their  recent  abilities  in  the  sub-domain  of  interest. 
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Appendix 

Design  Problem-Solving  Episodes 


Design  Episode 
Assimilate 


Decompose 


Evaluate 


Implement 


List 


Patch 


_ Definition _ 

Assimilate  episodes  have  the  goal  of  bringing  mformation 
from  the  external  environment  or  the  long  term  memory  into 
the  design  state.  Usually  the  mformation  being  assimilated 
is  constraint  infonnation,  but  also  bring  into  the  design  space 
specific  design  proposals  or  strategies  from  their  long-term 
memory,  colleagues,  and  books  (Ullman,  Dietterich  & 
Stauffer,  1988). 

Decompose  episodes  have  the  goal  of  developing  the  state  of 
the  design  by  decomposing  a  part  of  the  design  into  lower 
level  entities  Decompose  episodes  are  distinguished  from 
implement  episodes  (described  below)  by  the  fact  that  they 
lead  to  an  abstract  and  non-unique  representation  of  the  state 
of  the  design.  Decompose  episodes  can  be  a  fimctional,  or  a 
structural  nature.  Structure  decomposition  includes 
recomposition  information,  which  does  not  need  to  be  present 
in  functional  decomposition. 

Evaluate  episodes  have  the  purpose  of  evaluating  the 
progress  of  a  plan,  to  see  if  continuation  along  the  path  chosen 
is  still  likely  to  lead  to  the  desired  solution.  If  the  plan  was 
implicit,  evaluate  episodes  simply  determine  whether  the 
su^ect  perceives  that  he/she  is  getting  closer  to  a  solution. 
Note  that  evaluate  episodes  are  different  from  verify  and 
simulate  episodes  in  the  sense  that  they  are  not  concerned 
with  the  correctness  of  actions  and  solutions,  but  with  the 
validity  of  the  goal  structure. 

Implement  episodes  have  the  goal  of  developing  the  state  of 
the  design  into  a  low  level,  unique,  and  agre^-upon 
representation.  Additionally,  the  representation  has  to 
correspond  to  a  possible  physical  realization  of  the  design. 
List  episodes  organize  infoimation  (often  obtained  in 
assimilate  episodes)  into  a  list  of  alternative  from  which  a 
selection  can  be  made.  List  episodes  can  be  extremely  brief, 
and  are  recognized  as  an  entity  mainly  to  be  able  to  recognize 
sequences  of  episodes  that  have  the  purpose  of  evaluating 
alternative  solution  to  a  sub-problem. 

Patch  episodes  have  the  goal  of  altering  previously  specified 
information  to  repair  a  conflict  between  a  previously  accepted 
design  proposal  and  a  constraint  or  a  conflict  between  two  or 
more  constraints.  In  other  words,  patch  episodes  take 
information  accepted  in  the  design  state  and  reconsider  it  in 
the  light  of  conflicting  constraints  (Ullman,  Dietterich  & 
Stauffer,  1988). 


Digital  Circuit  Design 
28 


Plan 


Record 


Select 


Simulate 


Specify 


Verify 


Planning  episodes  have  the  goal  of  developing  strategies  for 
how  to  proceed.  Planning  episodes  do  not  help  in  solving  the 
goal  of  designing  a  machine,  but  are  a  part  of  establishing  the 
goal  structure  necessary  for  the  solution  (Ullman,  Dietterich 
&  Stauffer,  1988).  Note  that  planning  is  often  an  implicit 
process  that  caimot  easily  be  recognized. 

Record  episodes  have  the  purpose  of  recording  information  for 
the  designer.  This  includes  making  drawings,  notes,  etc.  The 
goal  of  this  episode  is  organizing  information,  rather  than 
communicating  it  to  others.  When  drawings  are  made  as  part 
of  another  episodes  (e.g.,  simulate)  they  are  not  classified  as 
record  episodes. 

Select  episodes  are  usually  preceded  by  list  episodes  and 
have  the  goal  of  selecting  an  option  from  a  list  of  alternatives 
based  on  some  kind  of  comparison.  Evaluation  of  the 
alternatives  is  not  part  of  this  episodes,  but  the  comparison  of 
the  results  is. 

Simulate  episodes  have  the  purpose  of  analyzing  the  cvirrent 
state  of  the  design  in  order  to  obtain  information  that  is  useful 
for  comparing  the  current  state  with  the  design  requirements, 
previous  design  states,  or  design  alternatives.  This  operation 
can  be  achieved  by  various  means,  including  computer 
simulation,  ment^  modeling,  formal  calculation,  etc.  The 
comparison  is  not  part  of  this  episode. 

Specify  episodes  have  the  goal  of  incrementally  making  the 
design  decisions  that  bring  the  design  closer  and  closer  to  its 
desired  state  (fully  specified).  Sometimes  this  is  achieved 
by  executing  an  algorithnuc  procedure  to  transform  one 
representation  into  another,  but  this  is  not  always  the  case. 
Verify  episodes  have  the  purpose  of  comparing  the  current 
state  of  the  design  with  a  previous  state,  or  with  the  design 
requirements,  to  make  sure  that  a  certain  step  (or  set  of  steps) 
in  the  design  process  was  (were)  performed  correctly.  This 
verification  cem  be  achieved  by  several  means,  including 
replay  of  operations,  and  comparison  of  simulation  results. 
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Table  1 

General  Characteristics  of  Experts 

Characteristic  of  Experts 

Study 

Excel  in  their  own  domains 

Chase,  1983;  Johnson  et  al., 

1981;  Voss  &  Post,  1988. 

Perceive  large  meaningful  patterns 
in  their  domains 

Akin,  1980;  Chase  &  Simon, 
1973;  Egan  &  Schwartz,  1979; 
Lesgold  et  al.,  1988;  McKeithen, 
Reitman,  Rueter,  &  Hirtle, 

1981;  Reitman,  1976;  Soloway, 
Adelson,  &  Ehrlich,  1988. 

Faster  than  novices  at  performing 
the  skills  of  their  domain 

Chase,  1983;  Centner,  1988. 

Quickly  solve  problems  with  little 
error 

Chase,  1983;  Centner,  1988. 

Have  superior  memory  for 
information  related  to  their  domain. 

Chase  &  Ericsson,  1982;  Egan  & 
Schwartz,  1979;  Reitman,  1976. 

Represent  a  problem  in  their  domain 
at  a  deeper  level  than  novices 

Allard  &  Starkes,  1991;  Berger 
&  Wilde,  1987;  Chi,  Feltovich, 

&  Glaser,  1981;  Weiser  & 

Shertz,  1983. 

Have  strong  self-monitoring  skills 

Chi,  1978;  Chi,  1987;  Chi, 

Glaser,  &  Rees,  1982;  Larkin, 
1983;  Miyake  &  Norman,  1979; 
Simon  &  Simon,  1978. 

Analyze  a  problem  qualitatively 

Lawrence,  1981;  Paige  & 

Simon,  1966;  Voss  &  Post, 

1988. 

Restructure  problems  more  often 

Akin,  1988;  Goel  &  Pirolli, 

1989. 
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Table  2 

Standard  framework  for  solving  combinational  logic  design  problems 


Component _ Function 


Understand 

Read  problem,  plan  the  design 

Truth  Table 

List  all  possible  input/output  combinations  for  a 
circuit. 

K-ntap 

Graphical  representation  of  the  output  from  each 
input.  Generally  used  for  minimization. 

Boolean 

Generate  Boolean  expressions. 

Implement 

Implement  circuit  using  logic  gates. 

Evaluate 

Check  to  ensure  circuit  is  correct  with  desired 
functionality. 
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Table  3 

Table  of  Instructor’s  Skill  Ratings  for  the  Student  Subjects 


ID  Skill  Rating^  CoujseGrad^ 


RP 

5.0 

B  (75%) 

JB 

7.5 

A(83%) 

DT 

6.0 

A(80%) 

ES 

2.0 

F(withdrew) 

TS 

7.0 

A(84%) 

®  Expert,  the  iiwtructor  of  the  course,  used  a  10  point  scale  with  10 
indicating  expert  performance. 

^  Vanderbilt  University,  Electrical  Engineering  285;  VLSI  Design,  class 
average  in  parentheses. 
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Table  4 

Example  of  list  episode  from  DT's  protocol. 


Segment 
segment  1 

segment  2 

segment  3 

segment  4 
segment  5 
segment  6 
segment  7 


_ Verbal  Behavior  (Non-verbal  behavior) _ 

Now,  since  it  is  a  NAND  the  gate  size  is  going  to  be  larger 
except. Cpoinfs  to  a  non-minimal  implementation  for  Borrow- 
Out) 

What  I  am  trying  to  think  is  if  I  can  include  any  of  these  two 
inputs  gates  into  part  of  their  third  input,  (points  to  the  two 
input  NAND  in  the  implementation  for  Borrow-Out  and  the 
three  input  NAND  in  the  implementation  of  Result  output) 
and  the  trade  off,  because  I  would  need  to  invert  the  output  of 
the  two  output  gate  to  bring  it  into  another  AND  gate  with  an 
input,  (draws  off  to  the  side,  the  implementation  of  the  gates 
as  he  describes  them) 

So  for  example  here  I  have  NOT  A  AND  B  (points  to  the 
Borrow-Out  implementation) 

and  here  I  have  NOT  A  AND  B  AND  NOT  BN,  (points  to  the 
Result  implementation) 

so  if  I  NOT  BN  AND  NOT  A  AND  B  at  this  point  we  have  got 
this  gatefadds  another  gate  and  labels  to  the  side  example) 
so  instead  of  designing  a  three  input  AND,  I  have  got  an 
inverter  and  a  two  input  AND,  (indicates  the  side  example) 
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Table  5 

Global  Plan  of  Expert  EM 


Segment 
segment  1 

segment  2 
segment  3 

segment  4 

segment  5 
segment  6 

segment  7 
segment  8 

segment  9 


_ Verbal  Statements _ 

Okay  so  what  I'm  going  to  do  is  going  to  write  a  table,  a  truth 
table  which  has  three  inputs.  Which  is  the  borrow  bit  and  two 
inputs  bit,  two  input  bits,  two  output  signals,  which  is  the 
output  borrow  bit  and  the  result  bit. 

From  that  you  are  going  to  try  to  uh,  find  a  function. 

Which  means  you  draw  two  Karnaugh  maps,  one  for  the 
result  bit  and  one  for  the  output  borrow  bit. 

You  are  going  to  put  down  the  ones  from  the  truth  table,  or 
you're  going  to  put  the  results  from  the  truth  table  in  the 
kamaugh  mar'  and  then  you're  going  to  uh,  find  the 
implicants.  .1  need  the  essential  prime  implicants  that  need 
to  be  in  ue  function 

and  from  that  you  should  try  and  write  down  the  function  in 
such  a  way  to  minimize  the  number  of  gates. 

You  might  uh,  come  up  with  two  functions,  one  for  the 
output  borrow,  one  for  the  output  result,  that  are  not  optimal 
by  themselves,  but  combined  could  still  result  in  lesser  gates, 
because  part  of  the  circuit  could  overlap. 

I  don't  Imow  if  that  will  be  the  case,  but  that  needs  to  be  seen 
then. 

Okay  from  the  functions,  well  from  there  it's  very  simple.  I 
mean  that's  just  the  mapping  from  the  functions  to  standard 
NAND  gates  or  NOR  gates  or  whatever  you  want. 

I  don't  know  which  of  the  two  is  easier  to  implement  in 
CMOS.  I  know  that  in  plain  TTL  the  NAND  gate  is  better,  I 
think,  but  you  are  sure  what's  cheaper  to  implement  in 
CMOS. 


Note.  Subject  has  a  blank  piece  of  paper  placed  next  to  the  problem 
statement.  Subject  does  not  begin  to  write  until  after  the  plan  is 
elaborated. 
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Table  6 

Solution  time,  episodes,  and  percentage  of  solution  that  involved 
Planning-Related  Activity  (PRA)  episodes 


Correct  Truth  Table 

Incorrect  Truth  Table 

EM 

JB 

DT 

AK 

TS 

RP 

ES 

Total  Time 
(minutes) 

45.15 

14.30 

37.15 

86.40 

30.00 

20.08 

94.30 

Rating 

ex 

7.5 

6 

ex 

7 

5 

2 

Total 

Number  of 
episodes 

50 

43 

79 

84 

62 

37 

99 

PRA 

1  pisodes 
(%  of 

Total) 

34 

55.8 

49.4 

29.8 

40.3 

32.4 

29.3 
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Table  7 

Example  of  interrelatedness  of  plan,  verifu.  and  evaluate  episodes  from 
AK's  protocol 

Segment  Verbal  Behavior  (Non-verbal  behavior) 

Episode _ 

1:  Plan  let's  verify  once  more,  (fust  finished  specifying  the  truth  table) 

2:  Verify  0,  0,  and  borrow  one  is  zero,  then  borrow  two  is  zero,  output 

one  is  zero  (pen  movement  indicates  line  1  of  the  truth  table) 
0,0,  in  the  borrow  one,  equal  one  means  you  rent  one  to  the 
lower  digit,  so  the  current  value  of  II  must  be  one,  so  one 
minus  zero,  is  one  B2  is  one.  (pen  movement  indicates  line  2) 
Okay,  II,  12, 0,1  and  borrow  one,  zero  means  you  meant 
(pen  movement  indicates  line  3) 

Okay  so  yeah,  where  was  I  maybe  somewhere  here. 

(finds  line  4) 

II,  0,0,1,  borrow  one,  is  zero,  then  you  borrowed  one  So  one 
oh,  (pen  movement  indicates  line  4) 

3:  Evaluate  this  zero  is  a  mistake  here  (pen  movement  indicates  line  3) 
if  you  didn't  loan  one  to  the  lower  digit  then  you  have  to 
borrow  one  from  the  upper  digit  one  minus  one  must  be  zero. 
(changes  0  to  1  in  output  of  line  3) 

II  is  0, 12  is  1  you  loan,  you  lent  one  digit,  one  to  the  lower 
digit.  So  this  is  one  it  is  one,  1-1  equals  zero  but  you  borrowed 
one  from  the  upper  digit,  (pen  movement  indicates  line  4) 

II  is  1, 12  is  0  then  B1  is  0,  if  it's  B1  is  zero,  you  don't  have  to 
borrow  anything  the  others  are  just  a  one 
(pen  movement  indicates  line  5) 

if  II  is  1, 12  is  0,  borrow  1  you  lent  one  to  the  lower  digit,  so  0-0 
is  0,  you  don't  have  to  borrow  anything 
(pen  movement  indicates  line  6) 

if  II  is  1, 12  is  1,  and  B  borrow  you  didn't  lend,  loan,  anything 
then  1-1  is  zero,  you  don't  have  to  borrow  one  from  the  upper 
digit,  (pen  movement  indicates  line  7) 

If  II  is  one,  12  is  1,  B1  you  lent  one  to  the  lower  digit,  then  the 
current  value  of  II  is  0,  so  zero  minus  oh,  zero  minus,  zero 
minus  one  is  one.  If  the  current  value  of  II  is  one,  and  uh 
okay,  (pen  movement  indicates  line  8) 

6;  Evaluate  this  is  wrong,  (indicating  output  of  line  8) 


4:  Patch 

5:  Verify 
continued 


(table,  continues) 
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7:  Patch 
8:  Plan 


this  is  one  (changes  the  0  to  1  in  output  of  line  8) 

okay  so  this  is  a  logic  table  now,  let's  go  to  the  NAND  gate 

level,  (finishes  with  truth  table  and  moves  the  paper  aside) 
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Table  8 

Distribution  of  Planning-Related  Activity  (PRA):  Across-Component 
vs.  Within-Component 


Correct  Truth  Table  Incorrect  Truth  Table 


EM 

JB 

DT 

AK 

TS 

RP 

ES 

Skill  Rating 

ex 

7.5 

6 

ex 

7 

5 

2 

Total  PRA 
Episodes 

17 

22 

39 

25 

25 

12 

29 

%  PRA  Across- 
Component 

65% 

50% 

15% 

20% 

52% 

33% 

21% 

%  PRA  Within 

35% 

50% 

85% 

80% 

48% 

67% 

79% 

Component _ 

Note.  No  skill  rating  for  the  experts. 
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